FIG. 1 portrays a common art apparatus to provide reflection scanning. In this figure, a reflection original, such as a paper document or reflection photographic print 102, is illuminated by a light source 104. A light path 106 from the light source 104 reflects from the print 102 as ray 108, and is focused by lens 110 onto a sensor 112. The sensor 112 typically may be a linear silicon sensor array such that, when focused by lens 110, it senses at any single time a line of points defining the scanning line 114 on the print 102. As time progresses, the print 102 is moved in direction 116 so that all points on the print 102 sequentially pass under the scanning line 114 and are sensed by the sensor 112.
The sensor 112 is attached by cable 120 to a computer 122. Associated with the sensor 112 are support electronics 124 that convert the analog signal from the sensor 112 into scan data which are digital numbers fed to the computer 122. Inside the computer 122 the scan data representing the image 130 is stored as a memory array consisting of an array of individual numbers 132 called pixels.
Typically, the sensor array 112 contains three lines, each line behind a filter of a different color, to scan three images simultaneously and produce multiple channels 133, 134, and 136 of the image, each representing a different primary color.
Also apparent in the scanned image 130 are defects such as dust, fingerprints, and scratches 138. These defects produce minor functional degradation in scanned images of documents; however, as reflection scanners are used more and more for scanning photographic images, these defects are emerging as a major limitation on the use of reflection scanners for this latter purpose. There are several reasons for these limitations. First, unlike a document which is primarily white or black and therefore requires only distinction between white and black, photographic images include all shades, and so even minor defects degrade the distinction between shades. Second, photographic images are very often much smaller than documents, typically five inches along each side, and therefore are very often magnified after being scanned. This magnification greatly increases the size and noticeability of defects. And thirdly, photographic images are often considered aesthetic works of art which are functionally degraded by small defects that would be ignored in a document scanned only for content.
Although a professional photographer might exercise more care for images and take special care when scanning them, the image literacy revolution is moving scanners into the hands of the general public and into publicly accessible kiosks, small office environments, homes, and schools. These environments are particularly prone to defective scans because the prints to be scanned are handled by people who are not professional image handlers. Accordingly, it is apparent that the automatic elimination of defects in a reflection scanned image would provide a major advance to the art and permit the image literacy revolution to move forward expeditiously.
FIG. 2 portrays a transmission scanning device 200 for inputting images from a transmissive media such as a negative film, or a positive film, sometimes generally called a transparency. A lamp 202 emits light ray 204 which transilluminates a film 206 and is received by a digital imaging device 208. The digital imaging device 208 may consist of a lens and linear sensor array as previously shown in FIG. 1, although many other configurations are commonly known in the art.
The digital imaging device 208 samples the brightness of the image at discrete points called pixels, turns each of these analog brightness measurements into digital numbers, and passes this data along cable 210 to computer 212. Inside computer 212 the image 214 is stored as a memory array 220 consisting of individual pixels 222.
The apparatus 200 may also include a filter wheel 226 containing several filters to color the light ray 204. For example, a specific filter 230 may color the light 204 red, and therefore provide a scan through camera 208 of the cyan dye in the film 206. Other filters can be used to capture images 230 and 232 in the other primary colors to give together a full color image. Other methods of distinguishing color are commonly known in the art.
Apparent in memory array 220 are dust, fingerprints, and scratches 236. In the past, these defects were a major problem for the industry. For publication images, some major magazines were able to seal the negatives in oil between glass to eliminate most of these defects, but such a solution is obviously not appropriate for the general public. Software designers have also attempted to solve the problem by selective softening, usually with extensive user intervention. As a result, most people working professionally with images have spent tedious hours manually removing these surface defects from images.
An advance in surface defect correction is taught in U.S. Pat. No. 5,266,805 issued to the present inventor. The theoretical motivation behind this prior art method is shown graphically in FIG. 3. In FIG. 3, the horizontal axis represents color arranged by wavelength, and the vertical axis represents brightness measured by transmission in a transmission scan or reflectance in a reflective scan. In this application brightness is referred to by the variable "x". The graph shows the brightness of the cyan, magenta, and yellow dyes used in photographic color images.
Under the wavelength of green light at 302, one sees the absorption of magenta dye as well as any surface defects. Under the wavelength of red light at 304, one sees the absorption of cyan dye and surface defects. Under the wavelength of infrared light at 306, an interesting thing happens; namely, all the dyes pass the infrared light and the image functionally disappears, so that under infrared light one sees a blank piece of film in addition to the surface defects. By dividing the measured red brightness by the measured infrared brightness, one can calculate what the measured red brightness would have been with no surface defects. After repeating this prior art process for all pixels in all primary colors, the surface defects can be erased from the image.
Returning to FIG. 2, an infrared selective filter 250 is added to filter wheel 226 and used in conjunction with digital imaging device 208 to provide a fourth color memory, or channel, array 252 consisting of individual pixels 254 each containing a number representative of infrared brightness at the corresponding point of the film 206. The infrared memory array 252 contains the defects 256 but no image 214 because the three dyes that create an image 214 in film 206 are all transparent to infrared light. Each pixel 222 in the visible memory array 220 is divided by the corresponding pixel 254 in the infrared memory array 252 by function 258 to yield a corrected pixel 260 in the corrected image array 262. This process is repeated for each color channel to produce the other color channels 264 and 266, automatically yielding a defect free image 268 from the film.
As will be apparent in the general description, the process of infrared surface defect correction as taught in the prior art was not extendable to reflection scans. This was unfortunate because on average significantly more reflection scans are made as compared to transmission scans. A method of extending infrared surface defect correction to reflection scanning would be a major advance to the art of digital imaging.
FIG. 4 illustrates a further related art that is background to the current invention. In the context of surface defect correction, infrared brightness can be used to determine the percent attenuation attributable to a defect. For example, if the infrared record were attenuated from 100% with no defect to 90% with a defect present, and the visible record transmitted 9% with the defect present, obviously the visible record would have transmitted 10% without the defect. A common mistake is to subtract, rather than divide, the infrared record. In the example just given, the infrared transmission dropped from 100% to 90%, a change of -10%. If -10% is subtracted from the measured 9% visible transmission, the result is a 19% transmission, a gross overcorrection.
Nevertheless, it is often desirable to perform a surface defect correction as a subtraction rather than a division. For example, it may be desirable to perform the correction separately on separate frequency bands. Obviously, one cannot divide two cycles that average to zero with any image remaining; one must instead use the linearity of a subtraction. To overcome this problem, the logarithm of each pixel is first calculated. A subtraction of two images, each in the logarithmic domain, is functionally equivalent to a division outside the logarithmic domain. In the example given above, log(100%)-log(90%)=log(10%)-log(9%).
The method of FIG. 4 receives a visible image 402 containing a defect 404, and an infrared image 406 of the same film with the defect 408. For reasons explained above, the logarithm is taken of the data in each pixel in the images to produce the log visible image 410 and the log infrared image 412. Further processing is then performed on small overlapping blocks of the images, such as block 416, enlarged as block 418 to show vertical strands of hair 420 and the horizontal scratch 422. Similarly, block 428 is enlarged as block 430 to show only the horizontal scratch 432. Next the visible block 418 and the infrared block 430 undergo a transform operation to yield the transformed visible block 436 and the transformed infrared block 438. The transform is selected to better isolate the defect scratch 422 from the hair 420. Although the hair 420 and the scratch 422 overlap each other in the visible block 418, they have different characteristics that can be used to distinguish them with the properly chosen transform. Such characteristics include angle and frequency. These characteristics are distinguished by several linear transforms, including the discrete cosine transform (DCT), and the discrete Fourier transform (DFT), both well known in the art. The DFT has the best separation of diagonal angles, but the DCT handles boundary conditions better.
In either the DCT or DFT, the vertical hair 420 produces a pattern 440 in transform space that is well separated from the pattern 442 produced by the horizontal scratch 422. The advantage of operating in transform space is now apparent: there can be more complete removal of the defect pattern 442 with less damage to the image pattern 440 if they have less overlap. This may be analogized to the increased ease with which a weed may be removed the farther it is from a flower.
In practice, the isolation of the resulting image and defect transform patterns will usually be less crisp than in this simple illustration; however, a transform such as the DCT will provide much better isolation than would be seen in the raw image.
Continuing with the related art method, the transformed defect block 438 containing the defect pattern 448 is split into two blocks 450 and 452, representing together a range between which the defect pattern 442 in the visible transform 436 is expected to lie. In particular, block 450 bounds the defect from below with a lowered defect limit 454, and block 452 bounds it from above with a raised defect limit 456. The range between defect limits 454 and 456 provides some "wiggle room" to ensure erasure of the visible defect pattern 442. Therefore, the infrared defect pattern 448 does not need to perfectly match the visible defect pattern 442 to insure complete defect removal. This "wiggle room" is critical to inexpensive scanners that may not image infrared details precisely the same as visible details. If the range is too small, not all the defect will be removed. If it is too broad, some image detail will be lost along with the defect. The better the transform is at isolating image and defect, the more the range can be expanded to ensure defect erasure without damaging image detail.
Function block 460 subtracts the defect blocks 450 and 452 from the image block 440 to produce the corrected block 462 containing, ideally, only the transformed image component 464. In particular, function block 460 is a smart subtraction that, for each element in block 436, will subtract whatever blend of the same matching elements in blocks 450 and 452 will give the smallest magnitude after subtraction. As an example, suppose specific element 470 of the image transform had a value of 10.0; the upper bound element 472 was 4.0; and the lower bound element 474 was 2.0. Obviously, subtracting 10.0-4.0=6.0 produces a smaller magnitude than 10.0-2.0=8.0, and so 6.0 is written into element 476 of the corrected transform image 462. Algorithmically, a trial subtraction is made of element 470 minus element 472, and of element 470 minus element 474. If the two results are of opposite sign, then the result of the smart subtraction 460 is set to zero; otherwise, it is whichever of the two has the smaller magnitude.
Finally, the inverse transform is taken of block 462 to yield corrected image block 480. This block 480 is placed back into the corrected logarithmic image 482 at block position 484. The process is repeated with all other, possibly overlapping, blocks in the image to construct the complete corrected logarithmic image 482. To finish, the antilog is taken of each pixel in image 482 to yield the finished image 486, similar to the input visible image 402 but without the defects 404.